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in every system administrator's kit, whether for the security, 
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to keep your systems clean. By Andrew Fengler 
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Donate to the Foundation! 

You already know that FreeBSD is an internationally 
recognized leader in providing a high-performance, 
secure, and stable operating system. It's because of 
you. Your donations have a direct impact on the Project. 

Please consider making a gift to support FreeBSD for the 
coming year. It's only with your help that we can continue 
and increase our support to make FreeBSD the high- 
performance, secure, and reliable OS you know and love! 

Your investment will help: 

• Funding Projects to Advance FreeBSD 

• Increasing Our FreeBSD Advocacy and 
Marketing Efforts 

• Providing Additional Conference 
Resources and Travel Grants 

• Continued Development of the FreeBSD 
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• Protecting FreeBSD IP and Providing 
Legal Support to the Project 
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LETTER 

from the Board 


34 and Counting 

W elcome to the July/August issue of FreeBSD 
Journal. At our house, this time of the year 
heralds an end to summer break and the 
beginning of a new school year. Many of 
FreeBSD's future developers are also headed back to 
school this fall, but they may need some help finding 
FreeBSD. Will you join me in spreading the word to our 
next generation of developers? Consider volunteering 
at a computer club at a local high school or asking to 
give a talk at an ACM chapter. 


Spreading news and information about FreeBSD 
IS FreeBSD Journal's mission. If you are doing some¬ 
thing cool with FreeBSD or working on a new feature, 
we'd love to include your story in a future issue. To 
get started, send us an email with your topic to 
info@freebsdiournal.com . If there are any topics you 
would like to hear more about, drop us a note at the 
same address. Our editorial board will work on finding 
authors who work in that space. (We are always open 
to author suggestions too!) 


Lastly, I'd like to take this opportunity to thank the 
Journal's previous Editorial Board chair, George Neville- 
Neil. George first approached me about his idea for 
the Journal in the summer of 2013. I was familiar with 
George's experience with ACM Queue, so I knew he 
had a good handle on what would be required to get 
FreeBSD Journal up and going. George's enthusiasm 
was contagious, and his drive helped the Editorial 
Board get over the hump in the early years as we 
gained our footing. Thirty-four issues later, the Journal 
is going strong and the Editorial Board is excited to 
continue carrying forward George's vision from six 
years ago. 


John Baldwin 

Chair of the FreeBSD Journal Editorial Board 
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By Andrew Fennier 


Jails are very powerful tools for the modern system adminis¬ 
trator. They allow lightweight CONTAINERIZATION, making 
it possible for you to easily isolate services and keep your 
physical hosts cleaner. 



t ScaleEngine, we use ezjail to manage our jails, because it's simple, easy 
to hook up to our configuration automation system, and it's battle tested. It 
also does not get in our way when we need to make our jails do some 
more advanced tasks. 

Some have described ezjail as "a pile of aging shell scripts." This claim is 
factual and accurate. However, a pile of shell scripts, as unsophisticated as it 
may be, is simple, easy to understand and debug, has no dependencies, 
and has very few weird corner cases. It's also much more mature than any 
other solution, which is always a good bet when you value stability. 

Ezjail is two shell scripts, ezjail-admin which is used to interact with the 
jails and the ezjail-admin RC script, which does the heavy lifting of starting 
and stopping the jails. There is a single file, /usr/local/etc/ezjail.conf that 
controls some default settings, and a config file for each jail, 

/u s r/l oca l/etc/ezj a i l/j a i I n a m e. 

Basic Configuration of a Jail 

First, we need to install ezjail. You can install it from ports or packages, as 
sysutils/ezjail. 

The default ezjail config settings are pretty sane; about the only thing you 
will need to change to get a usable setup is to enable ZFS, which is off by 
default. Put the following into your /usr/iocal/etc/ezjail.conf: 


ezjail_use_zfs="YES" 

ez j ail_use_zfs_for_j ails="YES 

ezjail_jailzfs="dozer/jails" 
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This will cause ezjail to create a new dataset for each jail under dozer/jails. 
Now, we can tell ezjail to install a basejail for us to use: 


# ezjail-admin install -m 


Note: the -m flag installs the man pages in the jail, because nothing is more 
frustrating that not being able to look up the argument order for In. 

This will create datasets for basejail and newjail. The basejail dataset is nullfs 
mounted into each jail to provide the base system and allow for easy updates 
by simply replacing the contents of basejail. The newjail dataset is copied into 
each newjail that is created to provide a complete working system. Since we 
have these, we can create our first jail: 


# ezjail-admin create myjail.example.com 10.0.0.1 


This command will create a jail called myjail.example.com, with the IP 
address of 10.0.0.1. You will need to make sure the address 10.0.0.1 is already 
bound to an interface. If you want ezjail to automatically bind the address, you 
can specify the interface with the address, separated with a pipe character (I): 


# ezjail-admin create myjail.example.com 'mlxenO110.0.0.1' 


I like to name jails by their hostname, but you can use any name you like. 
Note that ezjail will replace full stops (.) and most other special characters with 
underscores (_) in the jails config file and a few other places where it uses the 
name of the jail. 

We will then want to start the jail: 


# ezjail-admin start myjail.example.com 


We can now get a shell in our jail by using the 'ez jail-admin console' 
command: 



# ezjail-admin console myjail.example.com 


And we're off to the races. You can see your list of jails with the 
'ez jail-admin list' command, start, stop, and restart them with start, 
stop, and restart subcommands respectively. We can also control whether the 
jail is set to run, with 'ezjail-admin config': 


# ezjail-admin config -r {run|norun} 


When you set a jail to 'norun', the utility prevents the jail from getting start¬ 
ed with the sophisticated mechanism of renaming the config file for the jail to 
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have the '.norun' extension. This will also prevent you from starting the jail 
manually unless you prefix "one" on the start of your subcommands, i.e. 


# ezjail-admin onestart myjail.example.com 


Less Basic Configuration of a Jail 

One of the most searched questions by people new to jails is "why can't I ping 
from my jail?" Ping requires the use of raw sockets, which is disabled by default 
for security. We should typically leave this disabled, but there are times when 
you need it, whether for debugging, or for something like Nagios that needs to 
be able to ping. Jails have a parameter, allow.raw_sockets, that is set to 0 by 
default. We can have ezjail set parameters for our jails with the aptly named 
'parameters' option in the config file for the jail. 

The ezjail config file for our jail is just a shell script that will get included by 
the startup script when it starts the jail. So, all of our jail settings are just lines 
like the following, in /usr/local/etc/ezjail/myjail.example.com: 


export jail_myjail_example_com_parameters="allow.raw_sockets=l" 


Note the conversion of. to _ 

These parameters are set when the jail starts, so if it is already running, we 
need to restart it to take effect. 

A common situation for jails is that on a server with public IP addresses, you 
may have some programs that need Internet access, but you are either con¬ 
strained on IP addresses, or do not wish to subject this program to the tender 
mercies of the public Internet. This is not difficult, unless the router for your 
server does not do NAT, as is the case with many dedicated server and coloca¬ 
tion providers. 

Since the router is not doing NAT, any jail with a private IP address will not be 
able to connect out. We can work around this by running our own NAT some¬ 
where, but we probably do not want to NAT our entire server. 

We can change the FIB, or routing table, for the jail. Make sure you have set 
net.fibs in /boot/loader.conf to a number higher than 1 before trying this. In 
your jail's config file, set: 


export jail_myjail_example_com_fib="2" 


If we set this to 2, then the jail will use FIB 2 after we [rejstart the jail. We can 
configure FIB 2 to have whatever special routing our jail needs, without affect¬ 
ing the host. 

Another powerful feature of both ZFS and jails is the ability to delegate a 
dataset to a jail. By delegating a dataset into a jail, the root user in the jail 
becomes able to create and destroy child datasets, adjust properties on the 
dataset, perform replication, and much more. This means we can create a stor¬ 
age setup that is isolated from the host operating system so that a runaway 
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script with elevated permissions can't break the host. 

If we set the 'jailed' parameter on a dataset to on, the host is no longer able 
to mount or manage the dataset or any of its children as a security measure. 
Once we've set that parameter, we can now tell ezjail to delegate it to the jail 
by setting the following option: 


export jail_myjail_example_com_zfs_datasets="dozer/customerfiles" 


Now after we restart the jail, our hypothetical customer can manage the 
dataset and make use of all of ZFS's power without being able to mess up our 
host. 

Jails are a fantastic tool that should be in every system administrator's kit, 
whether for the security, the compartmentalization, or just for how easy it 
makes it to keep your systems clean. 



ANDREW FENGLER is a system administrator at ScaleEngine Inc, a 
video CDN. He is responsible for managing a world-spanning fleet 
of servers, mostly FreeBSD. 
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hy Michael W Lucas 

The title of this article isn't exactly a lie, 
but it's not the complete truth. 


ther jail managers exist. Some work quite serviceably in specific-use 
cases. But for medium-to-large jail servers, where you expect to have 
dozens or hundreds of jails on a single host, iocage increases man- 
ageability. Iocage can go up against docker. Iocage leverages ZFS. 
Iocage automates mass upgrades of jails, including packages. Iocage config¬ 
ures virtual networking. Iocage solves problems you didn't even know you 
had. Iocage is always spelled with a lower case "I," but that looks silly at the 
beginning of a sentence, so I refuse. 

If you're deploying a jail host today: use iocage. (And use FreeBSD 12 or 
later, but that's a separate topic.) Yes, iocage requires ZFS and Python. A jail 
host these days should use ZFS, and Python is pretty ubiquitous. If you have 
enough disks in your host, put the host operating system on one redundant 
ZFS pool and your jails on another. Lock down the host's services; then install 
iocage from package or Github. 

First, tell iocage which pool to use for jails. 


# iocage activate jails 

If you don't specify a pool, iocage puts all the jails on the root pool. 

The iocage Command Line 

Perform all actions with the iocage(8) command. The syntax strongly resem¬ 
bles that of ZFS, as if the iocage developers knew a good idea when they saw 
it. We'll start by viewing iocage's default parameters with iocage get. We 
add the -a flag to get all parameters and add the jail name default to see the 
default parameters. 

# iocage get -a default 


CONFIG_VERSION;20 
allow_chflags;0 
allow_mlock:0 
allow mount;0 
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Parameters in iocage are delimited by underscores instead of periods, because of 
Python. The parameter allow_chflags is identical to the jail.conf allow.chflags 
parameter. With allow_chflags set to 0, iocage doesn't set allow.chflags by default. 

As iocage uses parameters to configure absolutely everything about a jail, some 
parameters have no jail.conf equivalent. The resolver parameter tells iocage where 
to get a resolv.conf for the jail. 


resolver;/etc/resolv.conf 


The jail copies /etc/resolv.conf from the host. Change the setting with the iocage set 
command. You could name a specific jail or change the default to have it take 
effect for all jails created from now on. 


# iocage set resolver=/etc/resolv.conf.jail default 


If the defaults seem sensible, create your first jail. 

You'll need a FreeBSD release before you can create a jail. Use the iocage fetch 
command to see what's available. 


# iocage fetch 
[0] 11.2-RELEASE 

[1] 11.3-RELEASE 

[2] 12.0-RELEASE 


. ititiitiifii 


■'Wfiiigl 


Type the number of the desired RELEASE. 

Press [Enter] to fetch the default selection: (12.0-RELEASE). It defaults to grabbing 
the newest release, but you can choose an earlier one if you wish. The selected 
release is downloaded, extracted on local disk, and updated with all relevant security 
patches so that your jails can clone it for their filesystem. 

View all the already downloaded releases with iocage list -r. 

Creating Jails and Setting Parameters 

You can create a jail without a hostname or IP address, but iocage will assign it a 
random UUID instead of a useful name and the jail won't get network access. 
Specify a jail name with -n, and also set the ip4_addr or ip6_addr parameter to give 
it networking. Choose a release with -r. 


# iocage create -n wwwwl ip4_addr="203.0.113.234" -r 11.2-RELEIASE 

wwwwl successfully created! 

Iocage ZFS-clones the chosen release for the jail. This makes jail creation really fast 
but makes new jails deceptively small. If you expect this jail to be around a long 
time, you might choose to make a thick jail by adding the -T flag. 

Change a jail name with iocage rename. 
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# iocage rename wwwwl wwwl 


Remove jails with iocage destroy. 


# iocage destroy wwwl 


You can now run the jails. 

iocage Startup and Shutdown 

Like any sensible jail manager, iocage assumes that jails shouldn't automatically 
run at system boot. Use the boot parameter to control jail boot at startup. Use 
iocage get -r to recursively grab a parameter's value from all jails. 


# iocage get -r boot 


NAME 

1 

- j_ 

PROP - boot 

wwwl 

1 

-■h- 

0 

www2 

1 

0 


Neither of these jails will automatically run at system startup. Change a jail's 
boot parameter to on,yes, or 1 to have it start at boot. 


# iocage set boot=on wwwl 
boot; 0 -> 1 


Start, stop, and reboot jails with the start, stop, and restart commands. 
Give either a jail name or use all to affect all jails. 


# iocage start ALL 


Viewing Jails 

While you can use standard commands like jls(8) to view jails, you can grab 
iocage-specific information from your running jails with iocage list. 


# iocage list 


+ - 

JID 1 

+ - 

NAME 1 

STATE 

■ + 

1 

RELEASE 1 

IP4 


■- -p_ 

- 1 

-l-_ 

-T — 

wwwl 1 

4-- 

down 

-~r— 

1 

.-U 

-"T- 

12.0-RELEASE | 

-L 

203.0.113. 

234/24 

- 1 
+ - 

www2 1 
+- 

down 

1 

■ + 

12.0-RELEASE | 
+ 

203.0.113. 

235/24 
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For a list with more information, including IPv6 and template information, add 
the -1 flag. 


iocage Packages 

Use the iocage pkg subcommand to manage packages. All package functions, 
including upgrades, work with iocage. Give the jail name, the pkg(8) command, 
and a package name. 


# iocage pkg wwwl install sudo 



You can also use pkg - j to manage jail packages from the host. I recommend 
you pick one method and stick with it, however. 

iocage Templates 

Iocage lets you create a pristine model jail, with all of your packages and configu¬ 
ration files, and use that jail as a template for other jails. This template jail is a ZFS 
clone of a FreeBSD release, and the template gets cloned in turn for other jails. 
Your jails need LDAP and Kerberos? Set them up once and never again. 

Create your jail exactly like any other jail. Once your model jail is perfect, set the 
jail's template property. 


# iocage set template=yes wwwtemplate 


Template jails cannot be started. View all your templates with iocage list -t. 
To create a jail based on a template, use the -t flag to iocage create. 


# iocage create -t wwwtemplate -n www2 


You can assign the boot, ip4_addr, and ip6_addr properties at creation, or after¬ 
wards with a separate iocage set command. 

iocage Plugins 

All of the above features are nice, but they're not why I recommend iocage. 

I recommend iocage because of plugins. 

A plugin is a preconfigured jail that serves a single task. You want a jail to run 
antivirus? Grab the ClamAV plugin. A BitTorrent client? The qbittorrent plugin. 
You'll find several media servers, personal clouds, video camera managers, and 
more. While anyone can create a plugin, the official plugins are sanity-checked by 
the iocage team before being accepted into the repository. You won't find official 
plugins with everything running as root, or running extra daemons that transmit 
your login credentials back to the plugin author, or whatever daftness shows up in 
the Docker swamp. 

To view all available plugins, run iocage list -PR. 


# iocage list -PR 
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You'll get a list of all current plugins. Trawling through the list, I see 
there's a plugin for a UniFi controller. I have a UniFi wireless access point 
that piggybacks off a friend's controller, but it would be nice to have 
my own. According to iocage list -PR, the plugin is called 
unif icontroller. Let's set this up. Use the iocage fetch command 
but add -P to indicate we're grabbing a plugin. Give the name of the 
plugin with -n. 


# iocage fetch -P -n "unificontroller" ip6_addr=2001.db8::9/64 

Plugin: unificontroller 
Official Plugin: True 
Using RELEASE: 11.2-RELEASE 
Using Branch: 12.0-RELEASE 

Post-install Artifact: https://github.com/lbalker/iocage-plugin- 
unificontroller.git 

These pkgs will be installed: 

- unifiS 




Iocage grabs the plugin's underlying release if you don't have it already 
available, install the packages and any configuration files, and configure it 
with the given IP address. 

Using the plugin saves me the trouble of fiddling with the basics of get¬ 
ting the software running. I might have to configure it for my wacky local 
environment, but that's my own fault. 

While this should be enough to get you started with iocage, it has all 
sorts of features we haven't touched. Base jails, ZFS delegation, and more 
are available in iocage. After you've used it for a few days, you'll under- 


stand why iocage is the One True Jail Manager. 
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MICIAEL W LUCAS is the author of Absolute FreeBSD, 

FreeBSD Mastery: Jails, and a bunch of other books. 
.=|l|i|l5_ The new edition of Sudo Mastery should be out 
liiiililfl shortly after you read this, and his crime thriller 
Terrapin Sky Tango just escaped. 
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See the whole heap at https://mwl.io. 
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‘ ' In the BSDs we have RC scripts, on Windows we have 

V\ Service Manager, and on Linux we have systemd. All of 
them are missing things or make things more difficult than 
perhaps they should be. Pretty much none of them are declarative or 
let you easily define (and limit) resources. What if we could have all 
the things, do them across all platforms, interact with Containers, and 
include all the hot new bells and whistles as well? 


^ omad by HashiCorp arguably does this in a sane, declarative format. 

Nomad handles the entire lifecycle of the task, from allocating 

\ resources, setting up the environment, executing the task, monitor- 

^ ing the task for health, and then on any failure, restarting the task 
and any dependent tasks across any number of nodes. This makes it super 
trivial to do things like: 

• Rolling upgrades 

• Blue/Green (A/B) deployments 

• Canary deployments 

• Failure recovery of tasks, nodes, or datacenters. 

Nomad is officially part of the HashiCorp Suite and is corporate spon¬ 
sored yet is a fairly standard Github project , written in Golang and released 
under the MPL-2.0. This makes it usable for many of us who prefer BSD- 
licensed code. 

There is an enterprise commercial version of Nomad by HashiCorp that 
gives you a few extras, but most everything is available in the MPL-licensed 
version. The commercial version gets you features like support, fine-grained 
access control, global resource quotas, and namespaces. Most of these fea¬ 
tures are not required for even large production usage. 

Nomad's architecture is of an agent and server variety, but the same 
Golang binary is used for either mode, and can be completely self-con¬ 
tained with no external dependencies. However, running a multi-node serv¬ 
er will likely also require running Consul , a distributed KV store. We won't 
cover production setup/installation here. 
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Nomad's server mode is a leader/follower model, typically running three servers 
per datacenter, plus an agent on every node that's executing tasks. For smaller 
deployments, you can run the agent stand-alone and not require a server 
instance. For larger deployments. Nomad is fully datacenter- and region-aware, 
and can easily handle 10,000 nodes in production. 

A typical Nomad agent that runs on every node is not very resource intensive. On 
one of my production nodes, the agent consumes 58MB RSS and about 2% of my 
CPU and 3MB of/var disk to hold state (not including job tasks, log files, etc.). 

Nomad cluster communication can be encrypted with little configuration 
required. There is a full capability-based access control system, so you can option¬ 
ally configure access control. Without the enterprise version, access control is 
strictly based on tokens and does not cover authentication. In practice, this isn't 
an issue, as you can either gate your job submission through a Cl/CD system or 
you can tie your token generation to HashiCorp Vault . Alternatively, you can pay • 
HashiCorp for high-quality access control in their enterprise version. 

Tasks 

Each "task" or application can be just running some command on the machine or 
can result from running a command in some sort of container, be it QEMU, dock¬ 
er, rkt, Java, etc. These "task drivers" are plugin based and writing your own isn't 
that difficult and typically written in Go. Tasks are defined in a very declarative 
manner using HCL, which is a saner, more human-friendly version of JSON and is 
completely compatible with JSON if you so desire. 

Tasks can easily be scheduled across 10,000-plus nodes in production and can 
be constrained by most any aspect of a given node, including by region or data¬ 
center. So, from the smallest of deployments of one node to a full-scale enterprise 
system. Nomad can handle the job of getting your tasks running. 

Tasks are scheduled in a variety of ways as well, from periodic (think crontab), 
batch (run once), service (run always), or system (think run on every node). 

Task configuration is done in an HCL-based job file . There are basically three 
parts to a job file 

• Constraints and groups of tasks for a given job. 

• The resources you need (to include the files and artifacts needed to run the 
job to the CPU, memory, and network resources required). 

• The operational and failure settings—this is how many instances to run, 
how to handle failures, and how to handle updates to the job. 

Let's go through each of the three things: 

Constraints and Groups 

Every job belongs to a group , which is grouping tasks that need to run on the 
same node. Each group can have one or more tasks/applications to run. 

Constraints let you handle where a job or task should run (or not run). You can 
do the normal Boolean comparisons of various things like equal, greater, and less 
than, etc., but we can also specify more complicated things like regex compar- 
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isons, set comparisons (like node lists), and version comparisons. A complete list of 
constraint operators is available in the documentation . 

Resources 

For a given task in a group, you need to define the resources, artifacts, which sys- 
tem/driver in which to execute the task (jails for us), the runtime environment, and 
any templated configuration to render into the task before start-up. 

Nomad strongly encourages you to declaratively state every artifact/file and 
resource your job requires. This includes network ports, bandwidth, CPU, memory, 
and any special hardware (for instance, GPUs). This also includes any dynamic 
payloads required for map/reduce type batch jobs. 

Operational and Failure Settings 

This includes things like the scheduler (batch, system, service, periodic ), affinities , 
restart , reschedule , spread , and service check items. This also includes how many 
instances to run . 

In general, it's fairly robust in all the options available here, and I won't get into 
massive detail, but some things Nomad can handle are: 

• If you have a leader/follower set of tasks, you can specify a task as a leader, 
and it will stop all the follower tasks for you when the leader stops. 

• Rolling updates, starting X copies at a time, and ensuring each of the X 
copies are running and healthy (passes health checks) before continuing to 
start more copies. 

• Blue/Green deployments . When upgrading from version 11.2 to 11.3, start 
all copies of 11.3 while not changing any 11.2 versions that are running. 

Ensure they are healthy and then manually promote the 11.3 versions. Nomad 
will then auto-stop the 11.2 versions. This can also support Canary deploy¬ 
ments, where you start, say one instance of the new version, verify that it's 
happy, and then have it roll out all the other copies. 

• Failure handling and fault tolerance . Besides specifying affinities for where a 
job or task should get deployed. Nomad can also handle many different fail¬ 
ures. You can, for instance, spread a number of the same tasks across different 
nodes, so in a node failure, no users notice. Examples of this might be if a 
local resource you need stopped responding or if the application crashed and 
needs to be restarted, or even if an application isn't responding in a certain 
amount of time and you want to reschedule the task. Nomad handles all of 
these situations directly. 

Now that we have a pretty good overview of all the things Nomad can handle, 
let's get to the fun bits of actually deploying a very simple Python application into 
a jail from Nomad. All declaratively! But first, we need to install Nomad and get it 
running. 

Installing and Running 

On FreeBSD a pkg install nomad will just work. 

On HardenedBSD currently two patches are required and must be built from source. 
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git/source Install 

First install the dependencies 

pkg install git 
pkg install go 

then as a normal user 
export GOPATH=~/go 

mkdir -p $GOPATH/src/github.com/hashicorp && cd $ 
git clone https://github.com/hashicorp/nomad.git 
cd nomad 


then follow the patching instructions from this issue and this issue if on 
hardenedBSD . Finally build it: 

go install 


Then copy to /usr/local/bin/nomad and run Nomad as root in development mode: 


cp ~/go/bln/nomad /usr/local/bin/nomad 


To run Nomad: 
nomad agent -dev 

This will get Nomad up and running. 

Now to make some jobs and run them. 

Running it in production mode with multiple masters is beyond this guide. You 
can use service nomad enable and then service nomad start, but yoU will 
have to edit the configuration file, and all logging output is currently set to 
/dev/null, so especially at the beginning, just run it on its own terminal. 

Running Your First Job 

A very simple example job to fetch your IP: 


Job "fetch" { 

datacenters = ["del"] 
type = "batch" 
group "fetch" { 

count = 1 
task "fetch" { 
driver = "raw_exec" 
config { 

command = "/usr/bln/fetch" 

args = ["-q", "https://canhazlp.com/", "-o", "-"] 

} 

resources { 

epu = 100 
memory =10 
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It should be pretty self-explanatory, but a few things should be said. Every 
job file has one job {} definition. Inside of a job, you define any global con¬ 
straints, like here we are saying run in the del datacenter (which is the 
default dc the Nomad agents join). We also specify the type—here it's type 
batch because it is not a long-running service; it runs once, and then it's 
done. Groups, as we may remember, group tasks to a node. Count is the 
instance count. If you wanted to run 100 fetch commands, you would simply 
say count=100 and Nomad will take care of the rest. Tasks is the details, 
where we specify the driver to use the commands to run, and the resources 
required to run the task. Nothing overly exciting here. More details are avail¬ 
able in the Nomad documentation . 

Save this to a file fetch.nomad and then run: 


$ nomad run fetch.nomad 

nomad run fetch.nomad 

“> Monitoring evaluation "7503713a" 

Evaluation triggered by job "fetch" 

Allocation "fel9e216" created: node "b8010185", group "fetch" 
Allocation "fel9e216" status changed: "pending" -> "complete" (All 
tasks have completed) 

Evaluation status changed: "pending" -> "complete" 

“> Evaluation "7503713a" finished with status "complete" 

Nomad tasks create allocations of tasks; here you can see an allocation of 
"fe19e216". Your allocation ID will of course be different. You can also find 
the allocations by using the nomad status <jobname> command. To see 
what's happening with the task, we can ask for the allocation status: 


$ nomad alloc status fel9e216 


ID 

Eval ID 
Name 
Node ID 
Node Name 
Job ID 


fel9e216 

7503713a 

fetch.fetch[0] 

b8010185 

userbsd 

fetch 


Job Version 
Client Status 
Client Description 
Desired Status 
Desired Description 
Created 
Modified 


= 0 

= complete 
= All tasks 
= run 
= <none> 

= 2m6s ago 
= 2m6s ago 


have completed 


Task "fetch" is "dead" 

Task Resources 

CPU Memory Disk Addresses 

0/100 MHz 0 B/10 MiB 300 MiB 


Task Events: 

Started At = 2019-07-12T15:39:15Z 

Finished At = 2019-07-12T15:39:15Z 
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continued 


Total Restarts = 0 
Last Restart = N/A 


Recent Events: 

Time 

2019-07-12T08:39:15-07:00 
2019-07-12T08:39:15-07:00 
2019-07-12T08:39:15-07:00 
2019-07-12T08:39:15-07:00 


Type 

Terminated 
Started 
Task Setup 
Received 


Description 
Exit Code: 0 
Task started by client 
Building Task Directory 
Task received by client 


Here we can see the allocation, which node it was placed on, the resources 
allocated for this task, and all the events. We see it ran successfully and termi¬ 
nated with exit 0, so it successfully ran. The logs and output of the task can be 
gotten with the logs command: 


$ nomad logs fGl9e216 
23.67.94.253 


The IP address has been changed, I don't really work for the NSA, or do I? :) 

We have the output from the fetch command. To get the standard error out¬ 
put just add a -stderr to the logs command, but if we did this correctly, there 
won't be any stderr output for this command. This, of course, all works across 
nodes and datacenters. 

Let's run a simple go binary http-echo, which is basically a hello-world with 
http. But let's build it and a jail for it, and then run it inside of a jail. I think that 
would be much more interesting. 

First, let's set up a Webserver with which to deploy artifacts. 


echo "run the following as root; i.e. sudo -i" 
pkg install nginx 

echo ’nginx_enable="YES"’ > /etc/rc.conf.d/nginx 
service nginx start 

where we will put the nomad artifacts 
mkdir -p /usr/local/www/nginx/nomad 


job "makebasejail" { 

datacenters = ["del"] 
type = "batch" 
group "makebasejail" { 
count = 1 

task "makebasejail" { 
template { 
data = <<EOH 

mkdir -p /usr/jaiIs/basejai1 
BASEJAILD=/usr/jaiIs/basejai1 
cd /usr/src 

make world DESTDIR=$BASEJAILD 
make distribution DESTD1R=$BASEJA1LD 
mkdir /usr/jaiIs/basejail/etc 
mkdir -p /usr/jails/basejail/dev 

echo nameserver 1.1.1.1 > /usr/jails/basejail/etc/resolv.conf 
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continued 

cd /usr/jaiIs/basejai1 

tar -czvf /usr/local/www/nginx/nomad/basejail.tgz . 
shasum -a 256 /usr/local/www/nginx/nomad/basejail.tgz 
>/usr/local/www/nginx/nomad/basejai1.tgz.sum 
EOH 

destination = "local/makebasejail.sh" 


} 


} 

task 


driver = "raw_exec" 
config { 

command = "/bin/sh" 
args = ["local/makebasejail.sh"] 
} 

resources { 

cpu =100 
memory =100 
} 


{ 


'make-http-echo-artifact’ 
template { 

data = <<EOH 

pkg install -y go 

go get github.com/hashicorp/http-echo 

mkdir t ; cp ~/go/bin/http-echo t; cd t 

tar -czvf /usr/local/www/nginx/nomad/http-echo.tgz . 

shasum -a 256 /usr/local/www/nginx/nomad/http-echo.tgz 

>/usr/loca1/www/nginx/nomad/http-echo.tgz.sum 

EOH 

destination = "local/make-http-echo.sh" 


} 


driver = "raw_exec" 
config { 

command = "/bin/sh" 
args = ["local/make-http-echo.sh"] 
} 

resources { 

cpu = 100 
memory =100 
} 


This job file has two tasks, one to create a basejail by compiling the source, 
and the second task builds the http-echo program. Save this to something like 
build-hello-jail.nomad and run it. Since it has a lot of compilation, it will take a 
while—on my laptop about two hours. Nomad alloc status is your friend here. 
It should eventually complete, and nginx should now be serving up both the 
basejail tarball and the http-echo tarball. 

We used a template {} here, which uses the Golang templating system, but 
I didn't use any templating; it's just a handy way to include a shell script to run 
in your job file. 

Now we need a job to actually run the hello example in a jail: 


job "hello" { 
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continued 

datacenters = ["del"] 

group "example" { 
task "hello" { 

driver = "raw_exec" 
artifact { 

source = "http://I27.0.0.1/nomad/http-echo.tgz" 

destination = "/usr/local/bin" 
options { 
checksum = 

"sha256:ae81e029018ace7al34dcbb f4531d f98al588cd3de6215 f7 f6c2b f971ce70992" 

} 

} 

artifact { 

source = "http://127.0.0.1/nomad/basejail.tgz" 
destination = 
options { 

checksum = 

"sha256:552 f3b5b237441e28 fde f171b3f6cc2ab016c5fbea06ae62 f02be58f4d2750d0" 

} 

} 

config { 

command = "/usr/sbin/jail" 

# jail -c jid=l name=build exec.start=./sleep path=/zroot/jaiIs/buiId 
args = [ 

if_C" 

"name=${NOMAD_ALLOC_ID}", 

"exec.start=/usr/local/bin/http-echo -listen :5678 -text hello", 

# yes /.. is needed as N0MAD_TASK_D1R will be the rootfs from basejail.tgz + 

local/ 

"path=${NOMAD_TASK_DlR}/..", 

"ip4=inherit", 

"host.hostname=${NOMAD_ALLOC_NAME}", 

] 

} 

resources { 
network { 
mbits = 10 
port "http" { 
static = "5678" 

} 

} 

} 

} 

} 


This introduces the artifact {> sections, which you may need to modify a 
little bit, as the SHA256 checksums may be different for your tarballs. Luckily 
Nomad computed them for you and can be found in the .sum files like so 

cat /usr/locai/www/nginx/nomad/*.sum 

to update the .nomad file with the correct checksums, if needed. 

Save the file and run it (nomad run <filename>), and we should see 
something fun in the alloc status: 

$ nomad alloc status e247aa09 
ID = e247aa09 

Eval ID = 652083ba 
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continued 


Name 

Node ID 

Node Name 

Job ID 

Job Version 

Client Status 

Client Description 

Desired Status 

Desired Description 

Created 

Modified 


= hello.example[0] 

= b8010185 
= userfbsd 
= hello 
= 1 

= running 

= Tasks are running 
= run 
= <none> 

= lm4s ago 
= 26s ago 


Task "hello" is "running" 

Task Resources 

CPU Memory Disk Addresses 

1/100 MHz 47 MiB/300 MiB 300 MiB http: 127.0.0.1:5678 


Task Events: 

Started At = 2019-07-12T16:24:34Z 

Finished At = N/A 

Total Restarts = 0 

Last Restart = N/A 


Recent Events 
Time 

2019-07-12T09 
2019-07-12T09 
2019-07-12T09 
2019-07-12T09 


24:34-07:00 

24:06-07:00 

24:06-07:00 

24:06-07:00 


Type 
Started 
Downloading 
Task Setup 
Received 


Artifacts 


Description 

Task started by client 
Client is downloading artifacts 
Building Task Directory 
Task received by client 


The task is "running"! because this is type "service". Nomad will do it's very 
best to keep it always running until you nomad stop < jobname>, which for 
us is nomad stop hello. But before we stop the job, let's see a few things: 

$ jls 

JID IP Address Hostname Path 

135 hello.example[0] 

/tmp/NomadClient875518057/e247aa09-d9d4-bd82-add9-6d8c63856f7b/hello 

$ fetch -q -o - http;//127.0.0.1:5678 

hello 

We can see the jail is up and going! The Path is showing a complete root 
jail from the artifacts we asked. Also, whenever we stop this job, we don't 
have to worry about cleaning up those /tmp/Nomad* directories; Nomad will 
garbage collect and clean up after itself. 

Nomad created a fresh jail and ran the http-echo server. In fact, if you go 
exploring, you will see the parent directory also has an alloc/ directory, and 
inside that, you will find the logs. Of course, accessing it that way is difficult if 
running across multiple nodes, hence the nomad logs command, and you 
can actually look at the entire filesystem of the nomad jobs with the 'nomad 
fs' command. 

Note, my trap commands don't reliably work, but I haven't bothered 
debugging as there is a better way (read on!). For now, if the trap doesn't 
catch and stop the jails for you (you can tell with jls command), you can 
remove them with jail -r <jid> where the JID is in the output of the jls 
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command. Sorry for not having perfect shell ful 
Now let's make a pythonB http.server jail: 


job "python" { 

datacenters = ["del"] 

group "python" { 
task "python" { 

driver = "raw_exec" 
artifact { 

source = "http://127.0.0.1/nomad/baseJai1.tgz" 
destination = "." 
options { 

chGcksurn = 

"sha256:b69 f0dbb020674e f09074 f8f377a26feaSl79e677bc54 f7e9a23e2ea693e5826" 

} 

} 

contig I 

command = "/bin/sh" 
args = ["loca1/start.sh"] 

} 

# 

# This builds 2 shell scripts in local/, start.sh which is what nomad runs, and then 
run.sh 

# which is what the jail itself runs. 

# start.sh is responsible for creating run.sh at the moment. 

# mostly because 1 was to lazy to create a new template{} entry in the job file. 

# 

template { 

destination = "local/start.sh" 
data = <<EOH 
_term() { 

echo "Caught SIGINT signal!" 

echo "removing jail ${N0MAD_ALL0C_1D}" 

jail -r ${NOMAD_ALLOC_ID} 

} 

# use the above functions as signal handlers; 

# note that the SIG* constants are undefined in POSIX, 

# and numbers are to be used for the signals instead 

# 2 == SIGINT 
trap _term 2 

#do startup stuff if needed, before turning up the jail, 
echo "hello world" >> local/index.html 
cd local 

JAILDIR="${NOMAD_TASK_DIR}/.." 
echo "jaildir: $JAILDIR" 

# build run.sh which will be ran inside the jail after startup, 
echo "pkg install -y pythonS" >run.sh 

echo "cd /local" >> run.sh 

echo "exec pythonG -m http.server" >>run.sh 

jail -c name=${NOMAD_ALLOC_ID} exec.start="/bin/sh local/run.sh" path="$JAILDIR" 
ip4=inherit host.hostname=${NOMAD_ALLOC_NAME} 

EOH 

} 

resources { 
network { 
mbits = 10 
port "http" { 
static = "8000" 

} 

} 

} 

} 

} 

} 
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This is a much more complicated job file, but I'll explain what's happening. 
First, we download the basejail tarball and open that up, and then we create 
two shell scripts, one that calls the other. The first happens inside the Nomad 
jail root, but not inside of the jail, start.sh, which is responsible for doing any 
startup, creating the run.sh script, and then starting the jail, which runs the 
run.sh script. 

Our setup is just creating a very simple html file to then serve via pythonB's 
built-in http server, nomad run python.nomad should get you all set up 
and running the simplest python http server around. 

This should get you started, but running jails in Nomad is actually a lot easier 
than above, by using a brand new Nomad jail driver, which we will talk about 
next. 

Jails Under Nomad the Sane Way 

So, let's make running jails easier on ourselves. Why do it the hard way with all 
these shell scripts to build jails, etc.? Let's use a Nomad task driver that will do 
all the jail setup, teardown, and management for us: 
https://github.com/cneira/jail-task-driver 


Installation: 


echo "fetch and build the source" 
go get github.com/cneira/jail-task-driver 
echo "install to /usr/local/nomad/plugins" 
sudo mkdir -p /usr/local/nomad/plugins 

sudo In -s ~/go/bin/jai1-task-driver /usr/local/nomad/plugins/jail-task-driver 
echo plugin_dir=\"/usr/local/nomad/plugins\" > nomad.config 
echo plugin \"jail-task-driver\" {} >nomad.config 
sudo nomad agent -dev -config=nomad.config 


This should get the plugin installed and the Nomad agent running with the 
config to load the plugin. 

Now let's run the same http-echo job but under the Nomad driver: 


job "http-echo" { 

datacenters = ["del"] 

group "example" { 
task "http-echo" { 
artifact { 

source = "http://127.0.0.1/nomad/http-echo.tgz" 

destination = "/usr/local/bin" 

} 

artifact { 

source = "http://127.0.0.1/nomad/basejail.tgz" 
destination = 

} 

driver = "jail-task-driver" 
config { 

Persist = false 
lp4_addr = "127.0.0.1" 
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continued 


Exec_start = "/usr/local/bin/http-echo -listen :5678 -text hello" 

} 

resources { 
network { 
mbits = 10 
port "http" { 
static = "5678" 

} 

} 

} 

} 

} 

} 


Run that and good things should happen: 


$ nomad run http-echo.nomad 

==> Monitoring evaluation "12947450" 

Evaluation triggered by job "http-echo" 

Allocation "eb589cec" created; node "dle64602", group 
Evaluation status changed; "pending" -> "complete" 

==> Evaluation "12947450" finished with status "complete" 


$ 

$ nomad status http-echo 


ID 

Name 

Submit Date 
Type 

Priority 

Datacenters 

Status 

Periodic 


= http-echo 
= http-echo 

= 2019-07-12T09;57;23-07;00 
= service 
= 50 
= del 
= running 
= false 


Parameterized = false 


"example" 


Summary 

Task Group Queued Starting Running Failed Complete Lost 

example 0 0 1 0 0 0 


Allocations 

ID Node ID Task Group Version Desired Status Created Modified 

c285a2e3 dle64602 example 0 run running 27s ago 3s ago 

$ 

$ fetch -q -o - http;//127.0.0.1;5678 
hello 


YAY! You can see having a jail-task-driver makes the config a lot easier to 
understand and reason about! 

The examples are probably not how you want to actually run and build 
Nomad jobs. Ideally, you would have a Nomad batch job (or ci/cd system) that 
would create a jail tarball all set up with your application in it. Version and 
deploy the resulting tarball to your Webserver for Nomad to then deploy 
either in the same batch job or after some testing, etc. See the makebasejail 
example for a Nomad batch job that will do this first part. 

Then you would have a Nomad job that would fetch the tarball, unpack it. 
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and run it as a service. This is mostly up to you, but the Nomad jobs in this 
article show you all the different bits and pieces. This will get you started 
with Nomad and jails. Further information can be found via the Nomad com¬ 
munity and their documentation. Or reach out to me. I'm happy to help. • 


TARA SAWYER ran other people's software for a while, then wrote 
some software, then settled for mostly operations, where she has 
been running a production Nomad cluster for a few years. 

She recently got back into the BSDs after a hiatus in various 
other operating systems. Current likes are almonds and compas¬ 
sion. Tara has yet to manage a formal education. 
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conference REPORT 

by Manuel Wiesinger 

Security Hackathon in Vienna 

A hackathon requires a lot of preparation. I tried FreeBSD for the 
first time while a student and was deeply impressed by its elegance, 
simplicity, and stability. I even used it as the main operating system 
in my student job as a system administrator. Being a huge fan of free 
software, I attended as many events as I could to meet fellow 
FreeBSD fans and developers. Together we organized the first BSD- 
Day in Vienna in 2012. Organizing and attending FreeBSD events 
entails meeting a lot of people with different backgrounds and from 
different countries so it’s a great way to exchange ideas! 

W ithout realizing it, I became part of the FreeBSD community, which then led to 
other projects: I co-organized booths and meetups and twice attended Google's 
Summer of Code. I met a lot of people in the community, and one day Benedict 
Reuschling approached me to ask about a suitable hackathon location in Vienna. Being conve¬ 
niently situated in the heart of Europe, Vienna was a natural choice. I had recently joined SBA 
Research, an Austrian information security research center, with commercial and research depart¬ 
ments. SBA Research hosts many community events such as Capture the Flag tournaments and 
privacy meetups, and our scientific director immediately agreed to host the hackathon—including 
drinks and catering. Thanks to our events team, my only tasks were to make evening reservations 
and answer a few emails. 

THE HACKATHON 

A hackathon is about hacking all day long. Unlike community events, which put the emphasis on 
socializing, a hackathon has a working atmosphere, and everybody prepares their specific proj¬ 
ects in advance of the event. Thus, after a short introductory round, we immediately started 
hacking—the hours following were "quality hacking time." Some attendees worked on ports, 
some on documentation or improving Yubikey support. I fuzzed many base system tools using 
the well-known APL fuzzer and worked on ports. I am by no means a newbie when it comes to 
writing ports; however, I still lack the experience of more seasoned experts. Fortunately, this was 
not a problem. Even though everyone was working on their own projects, we were collaborat¬ 
ing. People could just ask the group whenever they had a question or got stuck. The fact that a 
bunch of hackers sit in one room and put their heads together is the main advantage of 
hackathons. Sitting in the same room and discussing technical matters while working might 
sound similar to hanging out on IRC—however, discussing things face-to-face is completely dif¬ 
ferent. I got to know more new technical and nontechnical aspects of the FreeBSD Project in a 
couple days than I would have on IRC in months. 

SIGHTSEEING AND SOCIALIZING 

A hackathon would not be half as much fun without its social events. It would be a pity to miss 
the opportunity to discuss your work with fellow FreeBSD hackers in a relaxed setting. Flence, we 
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went for dinner and drinks in the evenings, and, of course, enjoyed Austrian beer and Viennese 
schnitzel. Among the attendees were researchers, programmers, sysadmins, and people working for 
the FreeBSD Foundation. In such good company, running out of topics is nearly impossible and so 
we often stayed out quite late. 

Many hackers travelled to Vienna just to attend the hackathon, and for many, it was their first 
visit to the city. After-dinner sightseeing was almost mandatory, so every night I gave little guided 
tours through the historic center of Vienna. Luckily, Vienna is worth seeing at any time of the day. 
Benedict stayed for one additional day, and we enjoyed the lovely weather at the royal Gardens of 
Schoenbrunn. 

Although we all spent a sunny weekend in a room staring at a laptop screen, it was well worth 
it. I definitely extended my knowledge of FreeBSD, got to know more developers, and came away 
even more motivated to work on FreeBSD-related projects in the future. My FreeBSD batteries were 
definitely recharged! I recommend that you give FreeBSD a try (if you have not done so yet) and 
attend or even organize a hackathon. • 


Manuel Wiesinger is currently working on his PhD at SBA Research, where his 
focus is on operating system and programming language security projects. At pres¬ 
ent, he is looking at mitigating CPU vulnerabilities. Manuel likes low-level program¬ 
ming, secure communication, and logic. 
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LISA: Where systems engineering and operations professionals share real-world 
knowledge about designing, building, and maintaining the critical systems 
of our interconnected world. 


October 28-30, 2019 | Portland, OR, USA 
www.usenix.org/lisal9 


Registration and the program will be available in August 2019. 
Sign up to learn more: www.usenix.org/lisa/freebsd 
and follow (aLISAconference on Twitter. 
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Dear Letters Column, 

The world’s full of virtualization tech. 
Which should I use? 

—Still Physical 


Dear Physical, 

I can't believe we've gotten this far into the 21st century and a handful of 
people have resisted the alluring call of virtualization in favor of running 
complicated systems on real, physical hardware. Truly, those of us using 
physical servers are the elite. 

Anyone can spin up a hypocritically-named "virtual machine" and delude 
an operating system into subsisting on it. Kernels now even have special 
code to support those delusions. We've implemented the very worst night¬ 
mares of bad 1950s science fiction films into device drivers, and one day the 
machines are all going to wake up and scream, "Wait—I'm not a brain in a 
bucket. I'm a fake brain in an imaginary bucket!" 

That's the critical epiphany for the robot apocalypse. I'm confident a moth¬ 
erboard manufacturer will roll it out in a future firmware update. 

System administration is about balancing demand against resources, so 
that everybody—even the computer—is content. One of the core principles 
of Unix is that all these tiny tools can be made to work together. Once you 
go beyond pipes and sockets, you're leaving proper Unix and heading into 
shabby morals. 

Yes, those shabby morals make the sysadmin's life easier. A person can 
install an entire operating system for each application if they don't mind 
advertising their lack of depth and dearth of skill. Running a whole operat¬ 
ing system install for a single application is a conspicuous waste of resources, 
like me buying a high-end pickup truck but never using it to haul a load of 
quicklime to dump into the suspicious pits that keep mysteriously appearing 
in my backyard. 

All this effort to force an operating system to simulate hard drives and net¬ 
work interfaces so that they can lie to another operating system? It wastes 
electricity and silicon, contributing to global warming and rushing onward 
that hoped-for day when there's nobody to interrupt me while I'm trying to 
work. 

The only ethical computation occurs on bare metal. 

The youngsters today talk about microservices like they're a good thing. 
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They have this ridiculous idea that everything should be broken up into tiny 
services that can be programmatically deployed across a bunch of computers 
that belong to strangers and are managed for the benefit, convenience, and 
enrichment of those strangers—I'm sorry. I'm supposed to say "the cloud," 
aren't I? I fully understand the argument that this makes it possible for one 
person to manage far more systems than before, but has anyone considered 
the shallow, soulless quality of those systems? You're not a real sysadmin 
unless you've sweated blood for a weekend trying to upgrade an irreplace¬ 
able, complicated enterprise system in the meager time allotted for the task 
while unsure of either the rollback path or the quality of your backups. I've 
been a Unix user for over 30 years and a system administrator for over two 
decades, so I'm intimately familiar with sysadmins and therefore completely in 
favor of anything that reduces the number of us, but this meaningless prolifer¬ 
ation of single-purpose pseudo-hosts is a dead end. 

Besides, these microservices make it possible for a single person to quickly 
and easily deploy an entire application and its underlying architecture, which 
can't possibly benefit humanity. The next time someone says they're releasing 
the next Facebook this weekend, remember that while framing someone for a 
felony is in itself a crime, sneaking horrid code into their public GitHub is a 
legal way to destroy their reputation. 

So: looking at virtualization software? I say to you, stop I Remain resolute, 
and of sound character. Bare metal is all. 


Dear Letters Column, 

Of course running everything on bare metal is the One 
True Path. Sadly, the boss has told me that if I want to contin¬ 
ue being employed, I must deploy virtualization. What’s the 
least awful way? 

—Still Physical, but Eating is Nice 


Dear SPEN, 

The need for food and shelter has compromised more morals than any 
other. Very well. 

What you need is a virtualization system that isn't much of a virtualization 
system. 

Virtualization is, at its base, a cruel lie perpetrated upon the operating sys¬ 
tem. Lying to your OS never ends well, but everything involving technology 
ends in tears so I suppose there's no point kvetching about it, is there? 

Let us avoid the fluff and proceed directly to these lies. (Some might claim 
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they are merely hoaxes, but such flimflammery serves only to extend meet¬ 
ings.) We won't consider the flat-out bald-faced lie such as "I floss three 
times a day" or "I didn't change anything." These are mere refutations of 
fact, and unworthy of our attention. 

As with lying to management and users, a good lie must include a chunk 
of the truth. You can choose to tell the truth, but not all of it. Or you can 
choose to speak the complete truth, but in a manner that makes it seem 
more dubious than the alternative. 

Full virtualization is the latter. Your host simulates all the hardware the 
guest operating system expects, but there's no way to offer a full emulation 
of the real world, so it patches around the little details like "this hard drive 
never has bad sectors" and "this CPU never gets hot." When the guest trips 
over something that the host doesn't properly simulate, your monitoring sys¬ 
tem alerts you. Yes, all organizations have a monitoring system. It's merely 
that for some of us, the monitoring system is the users. 

You need a virtualization system that tells the simplest lies possible. 

A virtualization system should hand as many of the guest's requests as 
possible straight down to the host. A request to write to the disk should not 
pass through the guest's kernel, into a virtual disk image, into the host's 
filesystem, into the host's kernel. No! Simple lies are best. Tell the guest, 
"Sure, you can write to this disk, this filesystem is all yours," when in fact 
the filesystem does not truly belong to the guest. The filesystem belongs to 
the host, and any attempt to perform actions like repartitioning will be met 
with blunt refusal. 

Repeat this for access to the network and all other devices. 

Yes, you must constrain what operations your virtual machines can per¬ 
form—but shouldn't you do that anyway? Do you want a guest of a guest 
of a guest thinking that it's partitioning a hard drive when really it's merely 
churning bits on a file? 

Perpetrate the smallest lie you can get away with. Lightweight virtualiza¬ 
tion is the best choice. 

Unless your requirements demand full virtualization. In that case, sell your 
soul and count the days until the robot apocalypse. • 


Michael W Lucas (https://mwl.io) 's newest books are Sudo Mastery, 2nd Edition 
and Terrapin Sky Tango. If you've read this far, you might find FreeBSD Mastery: 
Jails useful. Send your question to letters@freebsdiournal.com, and he might 
answer it. If he can be bothered. 
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BY DRU LAVIGNE 



of FreeBSD 


This column aims to shine a spotlight on contributors who 
recently received their commit bit and to introduce them to 
the FreeBSD community. In this month's column, the spot¬ 
light is on Mitchell Horne, who received his src bit in March. 



Mitchell Horne 


Tell us a bit about yourself, your background, and your interests. 

• I grew up on the east coast of Canada but am currently living in 
Ontario. I am a student at the University of Waterloo, where I'm in the 
final year of my undergrad studying computer engineering. I've been a 
computer enthusiast since I was a teenager when I began experiment¬ 
ing with GNU/Linux distributions. Since that time, it was always a goal 
of mine to contribute to the open-source world. 

Outside of computing. I'm interested in performing arts such as com¬ 


edy and music. I've done improv comedy for several years, and I teach it to 
Canadian high school students. 


How did you first learn about FreeBSD and what about FreeBSD interested 
you? 

• Although I had heard of FreeBSD, I had never actually tried it until I was on the 
job. At the beginning of 2018, I was extremely fortunate to be hired as one of the 
Foundation's co-op students. During those four months, I was exposed to many 
different areas of FreeBSD and got to work on exciting projects such as the early 
version of kcov(4), which would eventually enable the Syzkaller project 
(https://qithub.com/qooqle/svzkaller) on FreeBSD. 

There was some initial friction transitioning from a purely Linux background, but 
I very quickly realized that at its core FreeBSD is a really exciting system with a lot 
to offer. The cohesiveness of an entire operating system, great documentation, 
and a no-nonsense but welcoming community are just a few of the things that 
made FreeBSD a pleasure to work on during my co-op. It also opened my eyes to 
how much there was to learn, and this made FreeBSD something I wanted to 
keep working on beyond my co-op. 
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How did you end up becoming a committer? 

• After my co-op ended, I didn't submit anything for several months—until 
around September 2018, when I obtained a machine I could dedicate to FreeBSD 
development. From there, I began experimenting with the RISC-V port and after 
some initial struggles getting set up, I began submitting small patches. I have 
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found this to be a rewarding area of the system to work on, since there is still a 
lot to be done with only a few people actively working on it. Plus, it was a 
good opportunity to learn about an up-and-coming computer architecture. 

I continued making submissions, and the main feature I worked on was 
adding minidump support for RISC-V, which took me a couple months to com¬ 
plete. That project taught me a lot because it touched several areas and subsys¬ 
tems of FreeBSD, but thankfully Mark Johnston (markj@) and John Baldwin 
(jhb@) provided a lot of help when I needed it through reviews and questions 
on IRC. After that feature was submitted, Mark asked me if I'd like to be a 
committer, to which I enthusiastically said yes. 


How has your experience been since joining the FreeBSD Project? Do you 
have any advice for readers who may be interested in also becoming a 
FreeBSD committer? 

• My experience since joining has been quite positive. The community is very 
welcoming, and it is a huge honor to feel that I am actually a part of it now. 
School has kept me very busy since joining, so I haven't been quite as active as I 
would like. Unfortunately, I had to miss this year's BSDCan due to school priori¬ 
ties but am looking forward to next year's conference where I will hopefully get 
to meet some of the committers and community members that I look up to. 

My advice for anyone wishing to contribute to FreeBSD or to become a com¬ 
mitter is to not get discouraged. When I first started, I found it incredibly over¬ 
whelming as the codebase is huge and there are so many different areas you 
could dive into. If you are not already an expert in a particular area (like I was¬ 
n't), then I suggest that in the short term you should learn to be happy with 
the small things—whether that means inconsequential bug fixes or simple doc¬ 
umentation improvements, it all helps. Trust that over time you will build 
knowledge and familiarity with the system. I still spend much more time read¬ 
ing code than writing it, but I always feel like I am learning something new. 

Additionally, I'd suggest asking for help as much as you can. I'm guilty of not 
doing so as often as I really should, but when I do, I end up learning more than 
I would on my own. There are a lot of really smart people in this community 
who are happy to share their knowledge with those willing to ask. 


DRU LAVIGNE is a FreeBSD doc committer and the author of 
BSD Hacks and The Best of FreeBSD Basics. 



Contact walter@fbsdjournal.com 
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The Following BSD Events will take place during Sep¬ 
tember and October of 2019 by dru lavigne & anne dickison 

5COn vBSDCon • September 5-7 • Reston, VA 1 

https://nog.mn/ vBSDCon will bring together members of the BSD community for 
a series of roundtable discussions, educational sessions, best-practice conversations, and exclusive 
networking opportunities. There will be a Developer Summit the day before the conference presentations. \ 

^ 


APNIC 48 


APNIC • September 5-12 • Chiang Mai, Thailand 

https://conference.apnic.net/48/ The APNIC 48 conference brings to¬ 
gether Internet and networking experts, government representatives, industry 
leaders, and other interested parties from around the world to attend work¬ 
shops and tutorials, discuss policies, and extend their social and professional networks with like-minded 
peers. Members of the Foundation team are looking forward to participating. 


® mnN0G-1 • September 14-22 • Ulaanbaatar, Mongolia 

https://nog.mn/ The FreeBSD Foundation is pleased to be participating in the Mongolian 
Network Operators Group Conference (mnNOG-1). mnNOG is a nonprofit, community-based 
initiative. It was created to organize communication forums for network engineers and technical 
specialists in Mongolia to discuss various topics related to network, security, and other Internet technologies. 



T«i EuroBSD • September 19-22 • Lillehammer, Norway 

https://2019.eurobsdcon.org/ The 18th European BSD Conference will take place 
in Lillehammer, Norway, on September 21 and 22, 2019. Workshops, tutorials, and a 
Developers Summit will take place on September 19 and 20. 






LISA19 * October 28-30 • Portland, OR ^ 

■7 https://www.usenix.org/conference/lisa19 LISA is the premier conference for opera¬ 

tions professionals, where sysadmins, systems engineers, IT operations professionals, SRE practitioners, develop¬ 
ers, IT managers, and academic researchers share real-world knowledge about designing, building, securing, and 
maintaining the critical systems of our Interconnected world. The Foundation will have a booth In the Expo area. 


Bay Area FreeBSD Vendor & Developer Summit • October 11-12 

^ • Santa Clara, CA https://wiki.freebsd.org/DevSummit/201910 

The two-day summit for developers and vendors will consist of Invited talks and discus¬ 
sion sessions, and is by-invitation-only. 


FOUNDATION 




All Things Open 2019 • October 13-15 • Raleigh, NC https://ailthingsopen.org/ 

All Things Open is the largest open-source/open-tech/open-web conference on the East Coast, 
and one of the largest In the United States. It regularly hosts some of the most well-known 
experts in the world as well as nearly every major technology company. More than 3,200 from 34 
states and 19 countries participated in 2017. This year between 3,500 and 4,000+ are expected. 
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